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(57) Abstract: This invention is a synchro- 
nization abstraction layer (SAL) providing 
a uniform interface to frameworks operating 
on sequenced flow data. It aUows content 
developers to design and build interactive 
content that will operate interchangeably in 
different multimedia frameworks (e.g., Apple 
Computer, Inc.'s (JuickUme™, Microsoft 
Corporation's NetShow™, RealNetworks, 
Inc.'s RealPlayer™, Sun Microsystems, 
Inc/s Java™ Media Framework) and on 
different hardware platforms (e.g., desktop 
PC. Macintosh™. Television set-top boxes 
such as those from General Instrument 
Corporation and Scientific Atlanta Inc., Inc.. 
Internet aj^liances such as AOL™-TV. and 
other appliances, e.g., a kitchen Internet 
radio). The uniform interface is independent 
of the particular framework and the platform 
on which the SAL is implemented, so that a 
single instance of content, whether created in 
Java^^, JavaScript, VBscript. HTML. XML, 
or some other language, can run appropriately on different hardware, e.g., on a Television set-top and on a desktop PC. In one 
realization, the synchronisation abstraction layer provides a Java™ VIRnrUAL MACHINE (JVM) interface for rurming Java™ 
plug-ins for streaming media apphcations such as Real Networks, Inc.'s RealPlayer™, Microsoft Corporation's Windows Media 
Technologies (NetShow™). Apple Computer, Inc.'s QuickTime™, Sun Microsystems, Inc.'s Java™ Media Framework. The JVM 
interface allows third-party developers to design platform- and framework-independent plug-ins for streaming media applications. 
This invention aUows content providers to use plug-ins or compatible software objects (such as Java™ applets) to build, for 
example, imeraclive streaming media content that is fully interactive but independent of the particular underlying hardware and 
software technologies, such as RealNetworks™ G2, Microsoft Corporation's NetShow™, a desktop PC, or a television. 
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For two-letter codes and other abbreviations, refer to the "Guid- 
ance Notes on Codes and Abbreviations" t^pearing at the begin- 
ning of each regular issue of the PCT Gazette, 
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CROSS-PIATFORM FRAMEWORK- INDEPENDENT 
SYNCHRONIZATION ABSTRACTION LAYER 

5 APPENDIX AND COPYRIGHT NOTICE 

This document includes a partial source code listing of a 
preferred embodiment of the applicant's invention. The code, 
listed in Appendix A and in the drawings, forms part of the 
10 disclosure of the specification. It is copyrighted. The copyright 
owner. has no objection to facsimile reproduction by anyone of the 
patent document, including the copyrighted Appendix A and the 
drawings as they appear in the Patent and Trademark Office file or 
records, but otherwise reserves all rights. 

15 

TECHNICAL FIELD 

This invention relates to the field of software plug-ins for 
multimedia file players and for other applications supporting 
20 ordered data flow files. More precisely, this invention defines a 
new field of software that allows plug-ins and content to be 
insulated from differences in underlying platforms and frameworks. 

BACKGROUND OF THE INVENTION 

25 

Multimedia player frameworks have become widespread. Later 
versions of the most popular Internet browsers - Microsoft's 
Internet Explorer and Netscape' s Communicator - include at least 
one player in the basic package. These are frameworks such as 
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RealNetworks, Inc.'s RealPlayer™ G2 family (collectively, the 
^RealPlayer™"); Microsoft Corporation's Windows Media Technologies 
(NetShow™); Macromedia, Inc.'s Director-; Apple Computer, Inc.'s 
QuickTime™; and Sun Microsystems, Inc.'s Java™ Media Framework 
5 (JMF). Most of these frameworks are extensible by means of plug- 
ins discussed below; some, notably JMF, are extensible. by means of 
applications built on top of the frameworks. By being extensible 
we mean that a particular framework supports a set of interfaces 
exposed for interaction with additional software modules or 

JO components. 

A framework is a set of interfaces, e.g., API's and 
procedures, and a set of capabilities exposed for the purposes of 
extensibility. It need not operate on multimedia-type files, i.e., 
files ordered sequentially and supporting the concept of a time- 
15 line; frameworks generally need not operate on time- or otherwise- 
ordered data files. In the rest of this document, however, we will 
discuss predominantly frameworks operating on ordered data flow 
files. We will refer to such frameworks interchangeably as 
"frameworks," "extensible frameworks," "extensible data flow-based 
20 frameworks," or by similar expressions; we do not imply any 
difference in the particular designation used, unless otherwise 
indicated . 

Ffameworks are extended by means of extensions, for example 
plug-ins. Plug-ins, also referred to as extension modules, are 

- 2 - 
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modules or components dynamically loaded at run-time. Extensible 
architecture and plug-ins have been used in commercial products, 
the RealPlayer^" and Windows Media Technologies being two instances. 
They also have been described in various publications. See United 
5 States Patents No. 5,838,906 and 5,870,559. (The specifications of 
these patents are incorporated by reference as if fully set forth 
herein.) A number of companies market plug-ins specifically for 
extending multimedia players. 

10 OBJECTS OF THE INVENTION 

One difficulty faced by plug-in developers is that a plug-in 
must be ported for each platform, i.e., for each hardware-operating 
system combination. Another difficulty lies in adapting a plug-in 
to various frameworks. Third difficulty, closely related to the 

15 first two, is that platform porting and framework adaptation 
require developers to have working knowledge of the various 
platforms and frameworks. One object of this invention is to 
provide a single interface that allows plug-in developers to build 
a single, platform independent version of a plug-in. Another 

20 object of this invention is to provide a uniform interface to the 
various frameworks so that a single plug-in can extend 
functionality of multiple frameworks. The third object of this 
invention is to facilitate development of content that can operate 
similarly, i.e., consistently, with different multimedia players 

- 3 - 
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u Tr,r- ' s RealPlayer™ and Microsoft Corporation' s 
(e g., RealNetworks, Inc. s Reaifxaye 

Windows Media Tachnolcies (NetShcw™, , . and platforms ,TV, PC. sat- 
top boxes, . The fourth object of this invention invention is to 
accelerate the develop,nent of content and „ulti.edia plug-ins by 
promoting reuse of existing software objects .e.g., objects written 
in Java™ and JavaScript) - 

QTTMMARY THE INVENTI ON 

This invention is an abstraction layer providing a uniform 
interface between a framework and one or more plug-ins. In the 
preferred embodiment, the invention is a Synchronization 
Abstraction Layer ,SAL, abstracting time-based frameworks into a 
conm,on synchronization interface. The SAL synchronizes itself and 
other plug-ins to a time-line of the underlying framework - and it 
does that independently of the underlying framework. In other 
words, the plug-ins interact with the underlying framework through 
the SAL, rather than directly. Typically, the SAL is implemented 
on top of the synchronization of the Application Progra^^^ing 
interfaces (API's) provided by the underlying frameworks. It has 
at least one point of coupling with the underlying framework: a 
method for providing the SAL with the current time. 

The SAL includes a uniform cross-platform interface for the 
plug-ins. preferably. the cross-platform interface has a 

functional core independent of the specific framework underlying 
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the SAL- 

Some of the features of this invention are listed below: 



1. Secure e-commerce , full interactive experiences including text 
5 and voice chat, games, graphics, animations, and advertising can be 

integrated and synchronized with streaming multimedia such as video 
and audio; 

2. It can be used to build a content framework that insulates 
10 content and plug-in developers from details and differences in 

hardware platforms, so that the same content or plug-in can run 
on desktop platforms (PC, Macintosh, Solaris, Linux), Televisions 
(GI, SA) , or other kinds of devices (AOL-TV, screen phones); 

J5 3, It can be used to build a content framework that insulates 
content and plug-in developers from specifics of, or differences 
in, software platforms, so that the same content or plug-in can 
run on Microsoft NetShow'^^', RealNetworks RealPlayer*^, Apple 
Quicktime'^, Sun Java^" Media Framework or any other media system; 

20 

4, It can run without a network or on different types of 
networks, such as wireless. Intranets, or the Internet; 

5. It can be used to synchronize arbitrary data types, 

- 5 - 
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including text, chat, graphics, video, audio, and active content 
like Java™, JavaScript, or Flash; 

6. The framework-independent layer can be implemented using 
5 different languages, including Java™, HTML, XML, JavaScript, 

VBScript, or other languages; 

7. When used for video synchronization, it can be used to 
synchronize arbitrary datatypes with different spatial objects in 

10 video, and with different temporal objects in video. 



15 



20 



TIRIEF PEgrRIPTIOU OF THE DRAWINGS 

Figure 1 is a diagram of basic processes in a traditional 
extensible time-based framework of prior art. 

Figure 2 is a diagram of basic processes in an extensible 
time-based framework incorporating the Synchronization 
Abstraction Layer in accordance with this invention. 

Figure 3 is a high level diagram of a preferred embodiment 
implementation specific to a table of contents plug-in. 

Figure 4 lists a sample toc.rj, an XML descriptor file for a 
table of contents plug-in of the previous Figure. 

Figure 5 lists a sample toc.xml, an XML descriptor file with 
detailed hierarchical description of the table of contents plug- 
in of Figure 3. 
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Figure 6 lists a sample toc.smi, an XML file defining a set 
of synchronized plug-ins. 

DETAIl^ED DESCRIPTION OF THE INVENTION 

5 

Figure 1 depicts conventional operation of an extensible 
multimedia player framework connected to a media server. Server 
110 includes Media Object 111, Clock Object 112, and Gustom 
Objects 113. Framework 120 and Extension Module (plug-in) 130 

10 coexist on the client side. As is usual for a media player. 

Framework 120 contains Video Media Object 121, Clock Object 122, 
and Audio Media Object 123. Extension Module 130 has Custom 
Objects 131 and 132, Server 110 connects to Framework 120 and 
Extension Module 130 through Network 150. Line 160 indicates the 

15 extensibility interface between Framework 120 and Extension 

Module 130. Note that this high level diagram of prior art omits 
many other components and processes, including an operating 
system and rendering means. 

Figure 2 demonstrates an arrangement functionally similar to 

20 the arrangement of Figure 1, but using the present invention. 

Server 210, including Media Object 211, Clock Object 212, and 
Custom Objects 213, connects to Framework 220, Extension Module 
(plug-in) 230, and Synchronization Abstraction Layer (SAL) 270 
through Network 250. Line 260 symbolizes the interface extending 

25 Framework 220 through SAL 270. Block 271 denotes Uniform 

- 7 - 
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interface between SAL 270 and various extension modules, 
including Extension Module 230. 

Bs before. Figure 2 omits many other components and 
processes, and is rather generalized. But a number of structural 
5 changes can be envisioned even in this, high level 

representation. For instance, media files can be stored locally, 
obviating the need for transmission over the networl.. Further, 
custom Objects need not reside on the same server as the media 
files; they may be located on a separate server or, in some 
10 cases, locally. Several extension modules can be ^simultaneously 
connected to the same client framework, either, through the SAL or 
directly. (The SAL itself is, of course, an extension module.) 

The discussion above focused on media players rendering 
multimedia files supporting the concept of time-line, such as 
IS video and audio clips, because these applications are 

commercially important. The invention, however, is not limited 
to rendering multimedia or time-line ordered files. Time-line 
sequencing is only one example of data flow order within files, 
including media files. Many other types of ordering are 
20 possible. AS one example, consider a video taken from a window 
of a car going down a highway. The images can be time- or 
distance-sequenced . 

Files of interest in the present context, which we designate 
as ordered data flow files, may also be ordered in multiple 
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dimensions. Imagine a video game called ''''Tank Commander," with 
the player controlling a virtual tank moving cross country. 
Because the tank can move in two dimensions, the top view of the 
tank and its immediate surroundings depends on two coordinates. 
5 The video data must therefore be ordered in two dimensions. 

Similarly, video of a one-directional view from an object moving 
in three dimensions requires three dimensional ordering. 
Clearly, the concept of ordered files goes beyond the four 
dimensions of space-time. 

JO We also need not limit this concept to files ordered 

according to variables perceived as continuous, e.g., space and 
time. The concept is equally applicable to ordering according to 
discrete variables, such as event occurrences or quantized space 
positions. As an example, consider a server storing a library of 

15 books and sending the content to the client for sequential 

viewing page by page. The concept of ordered data flow files is 
therefore a general one. 

Finally, in its general form the invention need not 
synchronize data flow, but may merely provide a uniform cross- 

20 platform interface for plug-ins. 

Although the invention can be implemented in various 
languages, it is preferable to use a platform-independent source 
code. Therefore, consistent with the spirit of the invention, 
its preferred embodiment is implemented in Java*^ - an 

- 9 - 
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architecture-neutral, object-oriented, multithreaded language 
intended for use in distributed environn^ents . (Partial source 
code of the preferred en^odiment is listed in Appendix A and 
constitutes a part of the disclosure of this specification.) The 
Java™ code is executed by a Java™ Virtual Machine (JVM), which 
is a byte-code interpreter. The JVM is stored in one location 
and instantiated separately for each extension module, 
eliminating the need to store duplicate copies of the JVM for 
other modules that are also implemented in Java™. 

Alternatively, the invention can instantiate a JVM from a copy in 
the framework, in the operating system, or even in another 
application . 

The preferred embodiment creates a uniform interface to the 
RealPlayer- of Real Networ)cs. It can be easily adapted to 
extend other multimedia players in addition to the RealPlayer™, 
especially if a con^non set of functional features exists in the 
various players. This should be the case for most multimedia 
players. Because the Java™ code of a specific extension can be 
mostly independent of a particular framework being extended, 

large sections of the SAL' s code can be shared among different 

framework adaptations of the same plug-in. 

Figure 3 illustrates a high level diagram of a more specific 

example of the preferred embodiment. Here, the SAL is used to 

plug-in a table of contents ("TOC") extension module into the 

- 10 - 
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RealPlayer^^ The TOC plug-in displays a hierarchical view of 
the table of contents of a video presentation described in the 
table of contents- In the hierarchy, entries in the table are 
highlighted during the video presentation of their associated 
5 video portions. A user can randomly access various portions of 
the video by clicking on the corresponding entries in the TOC. 

In this configuration RealPlayer*^" 301 has three extension 
modules: SAL (Real Java) 310, RealVideo 302, and RealText 303. 
(Hereinafter we denominate by '"rj" or "'RealJava" all Java"^" 

10 objects compatible with the G2 architecture.) For clarity, SAL 
310 is expanded within the dotted oval into its conceptual 
components - File Format Filter 311 and Renderer 312. This 
conceptual separation results from the G2 architecture 
constraints of the RealPlayer*^"; in a SAL adaptation compatible 

15 with another media player, such conceptual separation may be 
inappropriate. 

When a user invokes the RealPlayer"^" application and the TOC 
extension module, RealPlayer"^" 301 accesses Web Server 320 over 
Network 330 to retrieve ^^toc.smi,'' a synchronized multimedia 
20 integration language ('"SMIL") descriptor file defining a set of 
synchronized plug-ins to run simultaneously in the RealPlayer^ 
environment. This file contains a reference to ^^toc.rj," an 
ejctensible markup language (^"xml") descriptor file for the table 
of contents extension module compatible with the RealPlayer"^". 

- 11 - 
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Based on the MIME type of the toc.rj file. RealPlayer" 301 
instantiates SAL 310 and routes the toc.rj file to it. SAL 310 
includes File Format Filter 311 and Renderer 312, each 
i„,ple,nented using Java™ code linked with native platform 

code . 

File For„,at Filter 311 parses the toc.rj file and hands it 
over to Renderer 312 through a standard protocol defined by the 
RealPlayer'". The parsed data contains following elements: 
,1, name of the Java" class to instantiate, (2) location from 
which the aava" class can be retrieved; and (3) any other 
parameters specific to the extension module. A sample toc.rj 
file appears in Figure 4. The first line (starting with classic) 
specifies the Java" class names; the next two lines (codebar and 
archive, specify a URL location and names of Java™ class files; 
the following line assigns the «idth and the ..i^nt of the window 
for the table of contents; sync is the period in milliseconds 
between synchronizing events, i.e., between timing calls from the 
RealPlayer" to the SAL; duration refers to total length of the 
time-line of the presented event; and, lastly, P^ran, is the 
parameter that points to the URL with the tcc.xml file, the XML 
file descriptor of the specific table of contents. 

Renderer 312 instantiates OVM (Java™ Virtual Machine, 313 
needed to run the Java™ code, and also retrieves -.Jar" files 
(packaged Java™ class files, over Network 330. Note that the 

- 12 - 
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URL for the ^'.jar" files is available from the parsed toc.rj file 
discussed above. Renderer also instantiates an instance of 
RealTOC object 314 and hands to it the URL for toc.xml, the XML 
file describing the specific table of contents object associated 
5 with the multimedia presentation. This file includes the nodes 

and sub-nodes within hierarchical structure of the specific table 
of contents; it also includes the ^'begin" and "'end" times 
corresponding to each node and sub-node. RealTOC retrieves the 
toc.xml file, parses it, and builds the specific table of 
10 contents tree structure. A sample toc.xml file appears in Figure 
5. 

At the beginning of the multimedia presentation, RealVideo 
object 302 creates a video window for rendering video, while 
RealTOC creates a TOC window for rendering the table of contents. 

IS Both windows are rendered within the larger RealPlayer^^ window. 

As the time-line increases during the presentation, RealPlayer*^" 
301 periodically calls SAL 310 with the current time; SAL 310 
passes the current time to JVM 313, which in turn notifies 
RealTOC 314; RealTOC 314 highlights the appropriate node on the 

20 table of contents tree rendered in the TOC window. A similar 

process can be constructed with the SAL keeping current time and 
calling the RealPlayer"^ with time updates. 

When the user clicks on a particular heading within the 
table, RealTOC 314 sends the ^'begin" time associated- with the 

- 13 - 
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^ D^r.iPlHver™ 301 through JVM 
^r. i->ie ^^seek" function of RealPlayer 
heading to tne set^j^ 

3X3 and SA. 310.- KealPlayer™ 301 notifies all synchronized ™ed.a 
servers .not sho„n) of tne new -be,in" ti™e, waits for the all 
..e ™eaia servers to synchronize to the new ti.e. then updates 
, internal current ti^e and sends the new current ti.e to rts 

e.tension modules, i.e., RealVideo 302 and 310 .and RealText 

103 if applicable, ; SA. 310 updates the current ti.. of HealTOC 

through .odule 313. Importantly, both KealTOC 314 and 

RealVideo 302 are both synchronized to the current time of 
,„ RealPlayer" 301. Thus, after RealTOC 31. invoKes the see. 

^ - ^ current time does not skip to 

function requesting a new txme, .ts curren 

• ,n.il RealPlayer- 301 notifies RealTOC 314 of the 
the new time until RealFiaye 

Change in current time. If the see. function in RealPlayer 
is disabled, as it should be for live video, the time line 

75 continues uninterrupted. 

see. function is only one example of controlling the flow of 
.3ta rast forward and rewind functions also control the flow of 
data by chan.in. the rate of the flow. Moreover, as discussed 
previously, the data can be arranged in multiple dimensions,- 
then, its flow direction is a vector whose direction may be 
controlled from the extension module in a way similar to the 
control Of data flow from the table of contents described above. 
Mthough a specific e^odiment of this invention has been 
,5 Shown and described, it will be understood that various 
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modifications may be made without departing from the spirit of 

this invention. 
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I CLAIM: 

1. A method for extending an extensible framework 

comprising the steps of: 

providing the extensible framework; 

providing an extension module for the extensible framework; 

and 

proviaing an abstraction layer overlaying the extensible 
fra^eworR, the abstraction layer including a unifor-n cross- 
platforn, interface between the extension modnle and the 

extensible framework. 



2 A method for extending an extensible framework 
ecording to Claim 1, wherein the abstraction layer is compatible 

tensible framework and the uniform cross-platform 



a 

J5 with a second ex 

interface is framework- independent . 



3. A method for extending a first extensible ordered data 
flow-based framework comprising the steps of: 

providing the first extensible ordered data flow-based 

framework; 

providing an extension module for the first extensible 
ordered data flow-based framework; 

providing an abstraction layer overlaying the first 
extensible ordered data flow-based framework, the abstraction 

- 16 - 
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layer including a uniform cross-platform interface between the 

extension module and the first extensible ordered data flow-based 
framework; and 

synchronizing the abstraction layer to data flow in the 
5 first extensible ordered data flow-based framework. 

4. A method for extending a first extensible ordered data 
flow-based framework according to Claim 3, wherein data of the 
data flow is arranged in time progression. 

10 

5. A method for extending a first extensible ordered data 
flow-based framework according to Claim 4, wherein the first 
extensible ordered data flow-based framework is a multimedia 
presentation application, the method further comprising the step 

15 of rendering data-types included in the data. 

6. A method for extending a first extensible ordered data 
flow-based framework according to Claim 4, wherein said 
synchronizing step includes the step of the first extensible 

20 ordered data flow-based framework calling the extension module 
with current time. 

7. A method for extending a first er tensible ordered data 
flow-based framework according to Claim 4, wherein said 

- 17 - 
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synchronizing step includes the step of the extension module 
caUin, the first extensible ordered data flo«-based fra^ewor. 
with current time. 



20 



8, 



fST-c;+- extensible ordered data 
A method for extendxng a first extensi 

J- m^im 3 wherein data of the 

flow-based framework according to Claim 3, wne 

data flow is ordered in multiple dimensions. 

^ P'xtensible ordered data 

9 A method for extending a first extensiu 

flo«-based fra-newor. according to Claim 3. further including the 

step of the first extensible ordered data flow-based framewor. 

calling the extension module with current position of data. 

10 A method for extending a first extensible ordered data 
£lov,-based framewor. according to Claim 3. further including the 
step of the extension module calling the first extensible ordered 
data flow-based framework with current position of data. 

ai ft method for extending a first extensible ordered data 
flow-based framework according to Claim 3. further including the 
step of the first extensible ordered data flow-based framework 
calling the extension module with current rate of flow of data. 
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12. A method for extending a first extensible ordered data 
flow-based framework according to Claim 3, further including the 
step of the extension module calling the first extensible ordered 
data flow-based framework with current rate of flow of data. 

5 

13. A method for extending a first extensible ordered data 
flow-based framework according to Claim 3, further including the 
step of the first extensible ordered data flow-based framework 
calling the extension module with current direction of flow of 

10 data. 

14. A method for extending a first extensible ordered data 
flow-based framework according to Claim 3, further including the 
step of the extension module calling the first extensible ordered 

15 data flow-based framework with current direction of flow of data. " 

15. A method for extending a first extensible ordered data 
flow-based framework according to Claim 3, wherein data of the 
data flow represents discrete states. 



20 



16. A method for extending a first extensible ordered data 
flow-based framework according to any one of Claims 3-15, wherein 
the abstraction layer includes a mechanism for executing 
platform-independent code. 
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17 . 



A method for extending a first extensible ordered data 
flow-based framework according to any one of Claims 3, 5, 6, or 
7, Wherein data of the data flow is packetized and transported to 
the first extensible ordered data flow-based framework over a 
5 network, and the abstraction layer includes a mechanism for 
executing platform-independent code. 

18. A method for extending a first extensible ordered data 
flow-based framework according to any one of Claims 4, or 8-15, 
JO Wherein the data of the data flow is packetized and transported 
to the first extensible ordered data flow-based framework over a 
network, and the abstraction layer includes a mechanism for 
executing platform-independent code. 

J, 19- A method for extending a first extensible ordered data 

flow-based framework according to any one of Claims 3-15, wherein 
the abstraction layer includes a Java™ Virtual Machine. 

20. A method for extending a first extensible ordered data 
flow-based framework according to any one of Claims 3, 5, 6, or 
7, wherein data of the data flow is packetized and transported to 
the first extensible ordered data flow-based framework over a 
network, and the abstraction layer includes a Java™ Virtual 
Machine . 
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21. A method for extending a first extensible ordered data 
flow-based framework according to any one of Claims 4, or 8-15, 
wherein the data of the data flow is packetized and transported 
to the first extensible ordered data flow-based framework from a 

5 server storing the data over a network, and the abstraction layer 
includes a Java"^" Virtual Machine. 

22. A method for extending a first extensible ordered data 
flow-based framework according to Claim 21, wherein the first 

10 extensible ordered data flow-based framework is a RealPlayer. 

23. A method for extending a first extensible ordered data 
flow-based framework according to any one of Claims 3-15, wherein 
the abstraction layer is compatible with a second extensible 

J5 ordered data flow-based framework and the uniform cross-platform 
interface is framework-independent . 

24- A method for extending an extensible ordered data flow- 
based framework comprising the steps of: 
20 providing the extensible ordered data flow-based framework; 

providing an extension module for the extensible ordered 
data flow-based framework; 

providing an abstraction layer overlaying the extensible 
ordered data flow-based framework, the abstraction layer 
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including means for uniforn, cross-platform framework-independent 
interconnection between the extension module and the extensible 
ordered data flow-based framework; and 

step for synchronizing the abstraction layer to data flow in 
5 the extensible ordered data flow-based framework. 

25. Apparatus for using an ordered file through 
applications including an extensible ordered data flow-based 
framework, an extension module, and an abstraction layer having a 
,0 uniform cross-platform interface between the extensible ordered 

data flow-based framework and the extension module, the apparatus 
comprising: 

a computing machine running the extensible ordered data 
flow-based frameworlc, the extension module, and the abstraction 
15 layer; 

means for synchronizing the extension module and the 

extensible ordered data flow-based frameworks- 
means for providing content of the ordered file to the 

extensible ordered data flow-based framework. 



20 



26. Apparatus for using an ordered file according to Claim 
25, further including means for rendering data-types included in 

the ordered file. 
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27. Apparatus for using an ordered file according to Claim 
25, wherein the means for providing content includes a network 
connection and a first server storing the ordered file. 



- 23 - 



JNSDOCID: <WO 0079365A2_I_> 



wo 00/79365 



1 / 6 



PCTAJSOO/40279 




wo 00/79365 PCT/USOO/40279 

2/6 




^ O 

> -2 
^ -a 



o o 



3 : 



o 

§ 



O 

o -o" 
O 

(D 



O 

E .52 

1— I 



X 

—I 
< 

C/3 



c 
o 

o 
2 

CO 

< 

•2 

1 

c 
2 

o 
c 

CO 



o 



c: 



1 " 



o 



52 jo" 



6 
o 

GO 



< 

CO 



O 



E 
o 

CO 

=3 



< 



o 

J? 
O 



X 
o 

E3 



c 
c 



o 
X 

<u 

s 
s 

2 

txO 

o 



CO 

C 
X 



BNSDOCID:<WO 0079365A2 I > 



wo 00/79365 



3 / 6 



PCT/USOO/40279 





nrk70'5R c a o i •> 



wo 00/79365 



4 / 6 



PCT/USOO/40279 



Figure 4 
(toc.rj) 

<?xinl version="l . 0"?> 
<object 

classid="com. softcom. realjava.plugins . RealTOC" 

codebase=" . . / • . /website/demos/plugins/codebase/" 

archive="plugin -jar, sax . j ar , aelf red .jar" 

width="352" height="350" 

sync="1000" 

duration=" 900000" 

> 

<param naine="URL" value="toc.xinl"/> 
</object> 
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Figure 5 
(toe . xml) 

<?xml version-"1.0"?> 
<TOC> 

<NODE BE:GIN="Oms'' END="38684Tns"> 

<T ITLE>Ac knowl edgemen t s < /T I TLE> 
< /NODE> 

<NODE BEGIN-" 38 685nis" END=''57 627ins"> 

<TITLE>Body Teniperature</TITLE> 
</NODE> 

<NODE BEGIN="57628ins" END=="68999ms"> 
<TITLE>Exercise</TITLE> 
<NODE BEGIN="69000ins" END=" 96757ins"> 

<TITLE>Thennoregulatory System</TITLE> 
</NODE> 

</NODE> 

<NODE BEGIN="967 58ins" END="126114ms"> 

<TITLE>Course Overview</TITLE> 
</NODE> 

<NODE BEGIN="126115ms" END="319999ms"> 

<TITLE>Body Teniperature</TITLE> 
</NODE> 

<NODE BEGIN="320000ms" END=-323335ms"> 

<TITLE>Heat Related Illness</TITLE> 
<NODE BEGIN="323336ins" END- "37 17 8 3ms "> 

<TITLE>Heat ratigue</TITLE> 
</NODE> 

<NODE BEGIN="3717 84ms" END-" 39874 9ins "> 

<TITLE>Heat Cramps</TITLE> 
</NODE> 

<NODE BEGIN="398750ms" END="4 37021ms "> 

<TITLE>Heat Collapse</TITLE> 
</NODE> 

<NODE BEGIN=" 4 37022ms" END="510601ms"> 

<TITLE>Heat Exhaustion</TITLE> 
</NODE> 

<NODE BEGIN="510602ms" END'="560999ms"> 
<TITLE>Heat S1:roke</TITLE> 
<NODE BEGIN="561000ms" END="5908 9Bms"> 

<TITLE>Syinptoms</TITLE> 
</NODE> 

<NODE BEGIN=="590B99ms" END="654 966ms"> 

<TITLE>Treatment</TITLE> 
</NODE> 
</NODE> 
</NODE> 

<NODE BEGIN=" 654 967ms" END=" 900000ms"> 

<TITLE>Precautions</TITLE> 
</NODE> 

</TOC> 
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Figure 6 
(toe. smi ) 

<sinil> 

<head> 

<layout> 

<region id="videoRegion" left="0" top="0" 
width="176" height="120" 

fit="meet"/> 
<region id="tocRegion" left="0" top="120" 
width="176" height="200" 

fit="fill"/> 

</layout> 
</head> 
<body> 

<par> 

<video id=" video" 

src="rtsp: //demo, sof tcom. com/heat/video/heatsuresmooth. rm" 
region="videoRegion" /> 

<ref id="toc" src="toc.rj" region="tocRegion 
fill="free2e"/> 

</par> 
</body> 
</sinil> 



INSDOCID: <WO 0079365A2 I > 



wo 00n9365 



PCT/USOO/40279 



APPENDIX A 



/* SHeader: /cvs/ReaUava/src/rjavar/java/com/softcoin/realjava/plugins/RealTOCj 1.17 
1999/05/20 20:47:42 aw Exp $ */ 

// Copyright (c) 1998 SoftCom, Inc. AU Rights Reserved, 
package com.softcom.realjava.plugins; 

import javax. swing.*; 

import javax.swing.tree.*; 

import java.awt.*; 

import java.awt.event.*; 

import java.io.IOException; 

import java.net. URL; 

import java.net.MalformedURLException; 

import org.xml.sax.*; 

import com.microstar.xml.SAXDriver; 

import com.soflcom.realjava.*; 

import com.softcom.realjava.time.*; 

/** 

* Synchronized table of contents plugm. 

* Displays a table of contents (TOC) whose nodes are highlighted 

* in sync with the presentation. Clicking on a node seeks the 

* presentation to that nodes time. 

* <P> 

* < CODE >RealTOC< /CODE > imderstands a < CODE > URL < /CODE > param which 

* refers to an XML document specifying a hierarchical TOC. 

* <P> 

* Sample XML object element: 

* <P><TABLEBORDER=l><TR><TD> 

* <PRE> 

* <?xml version="1.0"?> 

* <object 

* classid = "com.soflcom.realjava.plugiQS.RealTOC" 

* archive= "plugin.jar,sax.jar,aelfred.jar" 

* width= "250" heights "235" 

* sync="1000" 

* duration^ "900000" 

* > 

* <param name = "URL" value = "toc.xml"/> 

* </object> </PRE> 

* </TD> </TR> </TABLE> <P> 

* This is the < CODR> TOC < /CODE > XML DTD. 
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* < CODE > BEGIN < /CODE > and < CODE > END < /CODE > are times specified 

* m the format documented in < CODE >TimeSpanRegistry.parseTimeO< /CODE > 

* Both are optional, but if either is specified, then both must be specified 

* <P> <TABLEBORDER=l><TR> <TD> 
5 * <PRE> 

* <!ELEMENT TOC (NODE+)> 

* &h; 'ELEMENT NODE (TTTLE, NODE*)&gt- 

* <!ATTUST NODE 

* BEGIN CDATA #IMPLIED 
10 * END CDATA #IMPLIED 

*> 

* <!ELEMENT TITLE (#PCDATA)«> </PRE> 

* </TD> </TR> </TABLE> <P> 

* Sample < CODE > TOC < /CODE > XML document: 
15 * <P> <TABLEBORDER=l><TR> <TD> 

* <PRE> 

* <?xnil version = "1.0"?> 

* <TOC«> 

* «<NODE BEGIN = "0" END= "4.999"> 

20 * <TITLE>Root node one</TITLE> 

* <NODE BEGIN = "5 " END = "9.999"> 
<TITLE>Node one child</TITLE> 
«<NODE BEGIN = "10" END="14.999"> 

<TITLE«fegt;Node one subchild l</TITLE«&gf 
25 * </NODE> . 

* <NODE> 

* <TITLE>Node one subchild 2</nTLE&gt- 

* </NODE> 

* </NODE> 
30 * </NODE> 

* </TOC> < /PRE > 

* </TD> </TR> </TABLE> <P> 

* RealTOC uses the AElfred XML parser and the SAX XML API 

* See <A TARGET="_top" 

35 HREF= "http://www.microstar.com/aelfi-ed.html" > http://www.microstar.com/aelfied.html < 
/A> 

* for information on the AElfred XML parser. 

* See <A TARGET="_top" 

HREF= "http://www.microstar.com/sax.htmI" > http://www.microstar.com/sax html</A> 
40 * for information on the SAX API to XML parsers 
*/ 

pubhc class RealTOC extends JScroUPane implements Phigin { 

private static fmal String MESSAGE_CATALOG = 
45 "com.softcom.realjava.plugins.RealTOCMessages"; 
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private PluginContext in_j)cContext; 
private JTree mJcTree; 

private TimeSpanRegistry m^^tsrRegistry = new TimeSpanRegistryO; 

private static final String DTD = "toc.dtd"; 

// Implements Plugin 
public Synchronized getSynchronizedO { 
return m_tsrRegistry; 

) 

// Implements Plugin 

public void startPlugin(PluginContext pc) { 
m_pcContext = pc; 

// Construct AElfred SAX driver 
Parser parser = new SAXDriverQ; 

// Set parser to handle the XML document, 
parser, setDocumentHandler(new TOCParserQ); 

String strURL = mj)cContext.getParameter("URL"); 
try { 

if (strURL = = null) 

throw new MalformedURLExceptionO; 
URLurl = new URL(m_pcContext,getDocimientBaseO, strURL); 
hiputSource is = new InputSource(urLopenStreamO); 

// Pass URL to DTD 

is.setSystemId(getClass0.getResource(DTD).toStringO); 

// Parse the docmnent 
parser.parse(is); 
} catch (MalformedURLException e) { 
Console.showConsoleO; 

System.err.printhi(MessageCatalog.getMessage(MESSAGE_CATALOG, 

getClassO, "msg.inf.invalidURL", strURL)); 
} catch (lOException e) { 

Console.showConsoleO; 

System.err.printhi(MessageCatalog.getMessage(MESSAGE_CATALOG, 

getClassO, "msg.inf.parse")); 

e -printStackTraceO ; 
} catch (SAXException e) { 
Console .showConsoleO ; 

Sy stem.err.println(MessageCatalog . getMessage(MESSAGE_C ATALOG , 
getClassO, "msg.inf.parse")); 
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Exception ee = e.getExceptionO; 
if (ee = = null) 

e.printStackTraceO; 

else 

ee .printStackTraceO ; 



10 } 



// Add the tree 

getViewport0.add(ni JcTree, BorderLay out. CENTER); 



15 



II Implements Plugin 
public void destroy PluginQ { 



// Object registered with event registry for each TOC event. 
// This object is also set as the TreeNode userObject 
private class TOCEvent implements TimeSpanListener { 
// Text to display in tree node 
2tf private String m strText; 

// -1 if no seek time specified 
private int m nTime = -1; 

II Path to node this event is associated with 
private TreePath m tpPath; 

public TOCEvent(DefaultMutableTreeNode tn) { 
^ m tpPath new TreePath(tn.getPathO); 

// Set text to display in tree node 
void setText(String strText) { 
m StrText = strText; 

35 } 

II Set time to seek media to tree node selected 
public void setTune(int nTune) { 
m nTime = nTime: 

40 } 

II Return seek time 
public int getTimeQ { 
return m nTime: 

} 
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// Event time reached, highlight tree node 

// Implements TimeSpanListener 

public void beginTimeSpan(TimeSpan ts) { 

// Select this tree node. This will make it visible too, 
5 mJcTree.addSelectionPath(m_tpPath); 
} 

// Implements TimeSpanListener 
public void endTimeSpan(TimeSpan ts) { 
10 II Deselect this tree node. 

m_jcTree.removeSelectionPath(m_tpPath); 

} 

// This is used by JTree to draw the node 
15 public String toStringO { 

return m strText; 

} 

}; 

20 private class TOCParser extends HandlerBase { 

// Root of tree 

private DefaultMutableTreeNode m tnRoot; 
// Current parent node 

private DefaultMutableTreeNode m_tnParent; 
25 // Current child node 

private DefaultMutableTreeNode m toCurrent; 

// Title text accumulator 

private StringBuffer m sbText = new StringBufferQ; 
30 private boolean m_bAccumuIateText = false; 

// Overrides HandlerBase 

public void startElement(String strName, AttributeList attrs) throws SAXException 

{ 

35 if (strName.equalsCTOC")) { 

// Create hidden root of tree 

m_tnRoot = m mCurrent = new DefaultMutableTreeNodeQ; 

} 

else if (strName.equals("NODE'')) { 
40 mtnParent = mtnCurrent; 

// Create a new node 

m tnCurrent = new DefaultMutableTreeNodeO; 

45 II Add node as a child of the current node 

mtnParent . add(m_tnCurrent) ; 
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// Create event for node 

TOCEvent te = new TOCEvent(m_tnCurrent); 
^ ni_tnCuiTent.setUserObject(te); 

// Set time on event if specified 

String strBegin = attrs.getVaIue("BEGIN")- 

if (strBegin != null) { 

String strEnd = attrs.getValue("END")- 
if (StrEnd == null) 
throw new 

SAXException(MessageGatalog.getMessage(MESSAGE CATALOG setaa.«A 
"msg.ex.nussingTime")); ~ ' 

15 try { 

int nBeginTime = TimeSpanRegistry.parseTime(strBegin)- 
mt nEndTime = TimeSpanRegistry.parseTime(strEnd)- 
// Set seek time in event 

te . setTime(nBeginTime) ; 
// Register event with the TOC timespan registry 
nEndTime)); ni_tsrRegistry.addTimeSpan(new TimeSpan(te. nBeginTime, 

} catch (NumberFormatException e) { 
throw new 

SAXException(MessageCatalog.getMessage(MESSAGE CATALOG getCIassH 
"msg.ex.mvalidTime"), e)- - * gei»-iasso. 

} 

} 

> 

else if (strName.equals("TITLE")) { 
// Reset String Buffer for new title 
m_sbText.setLength(0); 
^ m_bAccumulateText = true; 

^5 II Invalid element 

else 

throw new 

SAXException(MessageCatalog.getMessage(MESSAGE CATALOG getClassO 
"msg.ex.mvalidElement" , strName)) • " "^^^ getL-iassQ, 

} 

// Overrides HandlerBase 

public void endElement(String strName) throws SAXException \ 
if (strName.equals("TOC'')) { . 
// Finished building tree. Senip JTree with model. 
// Create the tree with the constructed nodes 
mJcTree = new JTree(m_tnRoot); 
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m JcTree. setRootVisible(faise); 

m JcTree. setShowsRootHandles(true) ; 

// Allow multiple selection so we can support overlapping timespans 

mJcTree.getSelectionModel0.setSelectionMode(TreeSelectionModel.DISCONTIG^ 
EE_SELECTION); 

// When a tree node is clicked (regardless of whether it was selected), 
// seek the video to the TOCEvent time registered with the node 
inJcTree.addMouseListener(new MouseAdapterQ { 

// XXX mouseClicked is unreliable - it is not always called (bugid 



4224704) 



e.getYO); 



//public void mouseClicked(MouseEvent e) { 
public void mouseReleased(MouseEvent e) { 

// Only handle single chcks 

if (e.getClickCountO != 1) 
return; 

// Get the path and node clicked on 

TreePathpath = mJcTree.getPathForLocation(e.getXO, 



if (path != null) { 

// Get the event registered with this node 
TOCEvent te = 

(TOCEvent)((DefaultMutableTreeNode)path.getLastPathComponentO).getUserObjectO; 

if (te ! = null && te.getTimeO > = 0) { 
// Seek the video 

m_j)cContext . seekPlay er(te . getTimeQ) ; 

} 

} 

} 

}); 

else if (strName.equals("NODE")) { 
// Back up a level 

mJnCurrent = (DefaultMutableTreeNode)m_mCurrent.getParentO; 

Lse if (strName.equalsCTITLE'')) { 
// Give title to current node 

((TOCEvent)m_mCtoent.getUserObjectO)^setText(m_sbText.toStringO); 

mbAccumulateText = false; 

) 

} 

// Overrides HandlerBase 

public void characters(char chO, int nStart, int nLength) throws SAXException { 
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if (mbAccumulateText) 

m_sbText.append(ch, nStart, nLength); 



}; 



/* 

// XXX debugging 

public static void main(String argsQ) { 
Frame fnn = new FrameQ; 
^ 0 frm . setLay out(ne w BorderLay outQ) ; 

RealTOC rt = new RealTOCQ; 
rt.startPlugin(new PluginContextQ { 
public URL getCodeBaseO { 
try { 

return new 

URU"file:S:/projects/realjava/src/rjavar/plugclasses/"); 

} catch (MalformedURUException e) { 
return null; 

20 } ^ 

public String getParameter(String strParam) { 
return "toc.xml"; 

} 

public int getDurationQ { 
return 0; 

} 

public void seekPlayer(int nTime) { 

public void showDocument(URL url. String strTarget) { 

}); 

fnn.add(rt, BorderLayout. CENTER); 
frin.setSize(300, 300); 
fiin.setVisible(trae); 

35 } 
*/ 
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